Determining Reliability of Subjective and Multi-label Emotion Annotation through Novel Fuzzy Agreement Measure
نویسندگان
چکیده
The paper presents a new fuzzy agreement measure γf for determining the agreement in multi-label and subjective annotation task. In this annotation framework, one data item may belong to a category or a class with a belief value denoting the degree of confidence of an annotator in assigning the data item to that category. We have provided a notion of disagreement based on the belief values provided by the annotators with respect to a category. The fuzzy agreement measure γf has been proposed by defining different fuzzy agreement sets based on the distribution of difference of belief values provided by the annotators. The fuzzy agreement has been computed by studying the average agreement over all the data items and annotators. Finally, we elaborate on the computation γf measure with a case study on emotion text data where a data item (sentence) may belong to more than one emotion category with varying belief values.
منابع مشابه
Inter-Annotator Agreement in Sentiment Analysis: Machine Learning Perspective
Manual text annotation is an essential part of Big Text analytics. Although annotators work with limited parts of data sets, their results are extrapolated by automated text classification and affect the final classification results. Reliability of annotations and adequacy of assigned labels are especially important in the case of sentiment annotations. In the current study we examine inter-ann...
متن کاملDetermining appropriate weight for criteria in multi criteria group decision making problems using an Lp model and similarity measure
Decision matrix in group decision making problems depends on a lot of criteria. It is essential to know the necessity ofweight or coefficient of each criterion. Accurate and precise selection of weight will help to achieve the intended goal.The aim of this article is to introduce a linear programming model for recognizing the importance of each criterion inmulti criteria group decision making w...
متن کاملFuzzy Neighbor Voting for Automatic Image Annotation
With quick development of digital images and the availability of imaging tools, massive amounts of images are created. Therefore, efficient management and suitable retrieval, especially by computers, is one of themost challenging fields in image processing. Automatic image annotation (AIA) or refers to attaching words, keywords or comments to an image or to a selected part of it. In this paper,...
متن کاملAn improved similarity measure of generalized trapezoidal fuzzy numbers and its application in multi-attribute group decision making
Generalized trapezoidal fuzzy numbers (GTFNs) have been widely applied in uncertain decision-making problems. The similarity between GTFNs plays an important part in solving such problems, while there are some limitations in existing similarity measure methods. Thus, based on the cosine similarity, a novel similarity measure of GTFNs is developed which is combined with the concepts of geometric...
متن کاملVerifying Music Tag Annotation Via Association Analysis
Music tags provide descriptive and rich information about a music piece, including its genre, artist, emotion, instrument, etc. While many work on automating it, at present, tag annotation is largely a manual process. It often involves judgements and opinions from people of different background and level of musical expertise. Therefore, the resulting tags are usually subjective, ambiguous, and ...
متن کامل